Information Theory and an Extension of the Maximum Likelihood Principle by Hirotogu Akaike

1. Introduction The problem of estimating the dimensionality of a model occurs in various forms in applied statistics. There is estimating the number of factor in factor analysis, estimating the degree of a polynomial describing the data, selecting the variables to be introduced in a multiple regression equation, estimating the order of an AR or MA time series model, and so on. In factor analysis this problem was traditionally solved by eyeballing residual eigen-values, or by applying some other kind of heuristic procedure. When maximum likelihood factor analysis became computationally feasible the likelihoods for diierent dimensionalities could be compared. Most statisticians were aware of the fact that comparison of successive chi squares was not optimal in any well deened decision theoretic sense. With the advent of the electronic computer the forward and backward stepwise selection procedures in multiple regression also became quite popular, but again there were plenty of examples around showing that the procedures were not optimal and could easily lead one astray. When even more computational power became available one could solve the best subset selection problem for up to 20 or 30 variables, but choosing an appropriate criterion on the basis of which to compare the many models remains a problem. But exactly because of these advances in computation, nding a solution of the problem became more and more urgent. In the linear regression situation the C p criterion of Mallows (1973), which had already been around much longer, and the PRESS criterion of Allen (1971) were suggested. Although they seemed to work quite well, they were too limited in scope. The structural covariance models of Joreskog and others, and the log linear models of Goodman and others, made search over a much more complicated set of models necessary, and the model choice problems in those contexts could not be attacked by inherently linear methods. Three major closely related developments occurred around 1974. Akaike (1973) introduced the information criterion for model selection, generalizing his earlier work on time series analysis and factor analysis. Stone (1974) reintroduced and systematized cross